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JOANN HSUER SS for expanding access to high-quality prekindergarten (pre-K) is at an a 


time high." Increased investments in early care and education have been spurred 
in part by rigorous evidence finding that four-year-old students who attend pre-K score 
higher on assessments of language, literacy, math, and executive functioning skills than 
children who do not attend pre-K.” Yet these initial positive impacts on cognitive and 
academic skills tend to diminish quickly after pre-K ends and disappear during kinder- 
garten or first grade.* Despite a smaller body of evidence finding positive impacts of pre-K 
into adulthood, this phenomenon has been described both as fadeout, when impacts 
of pre-K programs lessen over time, and as convergence, when the children who did not 
attend pre-K catch up to their peers quickly in elementary school.* These findings, coupled 
with recent results finding that students enrolled in a state pre-K program actually had 
slightly worse outcomes in sixth grade than students in a comparison group, have turned 
attention to the best ways to sustain pre-K gains.° 


Researchers have noted that redundant instruction across pre-K and kindergarten—when 
children learn the same skills and are exposed to the same instructional content in both 
years—is one possible explanation for the fadeout-convergence phenomenon. In response, 
a growing number of policymakers, practitioners, and researchers have proposed that ver- 
tical instructional alignment—or the implementation of standards, curricula, and assess- 
ments that build on one another as children move from pre-K to elementary school—is a 
key factor in supporting sustained impacts of pre-K.® Several states—including Califor- 
nia, Oregon, and Pennsylvania—have made pre-K to third grade (P-3) alignment a high 
priority in order to boost young children’s learning and development.’ Yet there is limited 
evidence showing that alignment does in fact promote better outcomes than high-quality 
instruction in general.® There are even fewer studies finding causal evidence on the bene- 
fits of policies that align instruction, and no studies that have used an experimental design 
to estimate impacts of a system-wide rollout of an instructional alignment approach across 
the early grades. As states begin to make large investments in P-3 alignment based on the 
idea that it is a best bet for supporting children’s long-term development, there is a clear 
need to conduct rigorous research to identify the scalable aligned approaches that most 
effectively bolster student learning. 


I ] Yaixe The Boston Public Schools (BPS) district is the first large school system in the United States 
SOILS UE REE ance to design and implement a curriculum and professional model—called Focus on Early 
. Learning—to align instruction across its well-known, high-quality public school pre-K 


program and early elementary school grades. Schools opt in and can choose how much of 


the curriculum to adopt. Further, BPS’s school assignment process combines information on parents’ 
preferences for where they would like their child to attend pre-K. When there are more students who 
prefer a school than there are available spots, this process uses naturally occurring lotteries—which 
approximate random assignment—to determine which students are assigned to each school. BPS’s 
implementation of Focus on Early Learning across its pre-K and kindergarten programs, coupled with 
this school assignment process, presents a unique opportunity to rigorously examine the effects of a 
district-wide rollout of instructional alignment on student outcomes across time. 


Although use of this methodology is an exciting prospect for research, early childhood studies that 
use naturally occurring lotteries like this are relatively new. To date, there are only a handful of 
papers that have used this method in an early childhood context, and one of those studies examined 
the effects of BPS’s pre-K program on children’s outcomes through third grade for the time period 
before the district undertook its alignment efforts. ° It is thus critical that researchers examining early 
childhood programs and policies carefully consider the opportunities and challenges of using this 
type of design before launching such studies. 


As part of the Expanding Children’s Early Learning Network (ExCEL) project, MDRC is partner- 

ing with with the Boston Public Schools (BPS) Department of Early Childhood, the University of 
Michigan, and the Harvard Graduate School of Education to understand how BPS’s reforms to align 
instruction across the early grades supported teaching and learning. The purpose of this brief is to 
assess the feasibility of using naturally occurring lotteries to evaluate the effects of BPS’s Focus on 
Early Learning approach on students over time; results from a full impact study estimating effects on 
student outcomes will be released in 2023. The brief begins by presenting background information on 
the approach, the motivation for the study, and the project’s core research aims. It then explores three 
issues related to the study’s feasibility that are critical for determining whether lotteries can be used 
to answer policy-relevant questions in a rigorous way: 


1. BASELINE EQUIVALENCE: Did the naturally occurring lotteries created by BPS’s school assign- 
ment process successfully produce two demographically equivalent groups—one assigned to 
schools choosing to implement Focus on Early Learning in pre-K and kindergarten and the other 
assigned to the business-as-usual control group—prior to the start of the pre-K year? 


2  GENERALIZABILITY OF THE STUDENT SAMPLE: To what extent were the students who were assigned 
to schools by these lotteries similar to or different from students who were not assigned to a 
school through a lottery? 


3 GENERALIZABILITY OF THE SCHOOL SAMPLE: To what extent were the BPS schools that chose to 
implement Focus on Early Learning similar to or different from those that did not? 


The brief concludes by discussing whether this study is successfully set up to draw causal conclusions 


about the impacts of a district-wide rollout of instructional alignment across the early grades while 
also producing results that are generalizable—and relevant—to the broader population of BPS pre-K 
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students and schools. Findings presented in this brief will help ensure that comparable efforts in 
the future have direct relevance for policymakers and practitioners. Results will also provide much- 
needed practical guidance for lottery-based studies of early childhood programs and policies. 


INSTRUCTIONAL ALIGNMENT AS A TOOL TO SUSTAIN THE PRE-K BOOST 


Aligning curricula—along with high-quality implementation supported by professional development 
for teachers—across pre-K and elementary school has emerged as a potential strategy for sustain- 

ing the pre-K boost into the early elementary years.'° Alignment has typically been conceived as 
encompassing standards, curricula, and assessments that build on one another over time (vertical 
alignment) and are consistent across schools and classrooms within a given grade (horizontal align- 
ment). The focus of this brief is vertical alignment, which policymakers prioritize when designing 
approaches to sustain pre-K benefits. A vertical alignment approach may look like this: Pre-K focuses 
instruction on foundational early math and literacy skills (for example, letter sounds and recogni- 
tion, numeracy, cardinality), while elementary school follows with sequentially more challenging 
tasks and concepts and continued exposure to developmentally appropriate learning formats. Recent 
work has argued that aligned content must also remain high quality over time to boost learning and 
development." 


Although the idea of building on children’s knowledge and skills over time may not seem particularly 
novel, research shows that existing kindergarten instruction often repeats content that children—and 
pre-K attendees in particular—already know." In addition, instructional learning formats are incon- 
sistent across these grades. Researchers at the University of Michigan, MDRC, and Harvard found 
that pre-K students in Boston spent more time engaged in learning centers—or structured, child-di- 
rected, play-based activities located in areas around the classroom—while kindergarten students 
spent more time in whole group instruction and working individually."* In a different study, MDRC 
researchers found that kindergarten students learning math spent 83 percent of their time in whole 
group settings while pre-K students spent 26 percent of time in those settings."* 


Even though access to pre-K has expanded during the last 15 years, kindergarten instruction and 
standards have remained largely unchanged.** In response, researchers, policymakers, and practi- 
tioners increasingly argue for the need to vertically align instruction across pre-K and elementary 
school so that students continually build on learning gains over time. And there are inklings that 
these reforms may help sustain the pre-K boost. MDRC researchers found that students who experi- 
enced an evidence-based math curriculum in pre-K and aligned math supports in kindergarten—in 
the form of math clubs taking place outside of typical instruction—had higher standardized test 
scores in third grade compared with students who did not.'® Yet the few rigorous studies of instruc- 
tional alignment completed to date have been implemented as demonstration studies, testing impacts 
under conditions controlled largely by researchers. Evidence on the impacts of instructional align- 
ment when implemented at scale by a whole school district is even more limited. 
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USING NATURALLY OCCURRING LOTTERIES TO ESTIMATE IMPACTS 
OF A DISTRICT-WIDE ROLLOUT OF INSTRUCTIONAL ALIGNMENT ON 
STUDENT OUTCOMES 


Estimating the causal impact of any scaled, district-wide policy can be quite challenging. Although 
randomized controlled trials (RCTs) are the gold standard for examining the effects of education 
programs and policies, mounting an RCT to test the effects of a district-wide instructional alignment 
approach across pre-K and kindergarten would be logistically difficult and very expensive. Increas- 
ingly, however, there are opportunities to use naturally occurring lotteries that exist within districts’ 
school assignment processes to take advantage of a random assignment mechanism implemented as a 
part of typical educational practice. 


The basic approach is as follows: In cases where there are more students who want to attend a given 
school than there are slots, a random assignment process is used by the school district to determine 
which students are assigned those slots. Depending on the district and assignment process, this 
approach is repeated until all the slots in all the schools are filled. In any RCT study using this type of 
data, program group students are those who “win” a spot in their preferred school, and control group 
students are those who “lose” a spot in their preferred school. The data from each of these lotter- 

ies can be combined to conduct one larger experimental study of a district-wide policy or practice. 
Because assignment is random, the program and control groups’ observed and unobserved charac- 
teristics at the time of randomization should be equivalent. Therefore, any difference in the outcomes 
of students in the program and control groups can be described as the causal impact of the program 
or intervention of interest. This type of design has been widely used to study interventions in K-12 
settings such as small high schools in New York City and charter schools.’” Yet this approach has 
only been used in a small number of studies of pre-K programs to date."® 


Naturally occurring lottery studies have several strengths. First and most important, researchers can 
use this design to estimate causal effects of a given program or intervention on student outcomes, 
creating a gold-standard research design. Available administrative data allow the team to confirm 
that the lottery “worked” by creating program and control groups that are equivalent along observed 
characteristics. Second, naturally occurring lotteries often provide the opportunity to study a pro- 
gram or policy across an entire school district or pre-K system, implemented in the population of 
interest. Lotteries are also useful from a practitioner perspective because they allow school districts 
to incorporate parent preferences into school assignments and allocate slots in oversubscribed 
schools fairly. Finally, lottery studies can be conducted with retrospective and prospective informa- 
tion, allowing researchers to examine multiple years of data to learn about the impacts of a given 
program across an extended time period. 


Yet there are also challenges and limitations associated with this design that researchers must 
address. First, there is no guarantee that a school district’s assignment process will be random and 
create comparable program and control groups. This means that researchers need to fully investi- 
gate the random assignment process and assess whether there is evidence that all characteristics of 
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the program and control groups are equivalent at the time of randomization. Second, lottery studies 
often reflect impacts of a policy on a unique subset of students who may differ from the broader pop- 
ulation. Students must compete for a lottery in an oversubscribed school to be included in a lottery 
study. Those who do not apply to oversubscribed schools will be excluded from the impact study; if 
students who apply to oversubscribed schools are different in some way from students who do not 
apply to those schools, then the results of the impact study will not generalize to the broader popu- 
lation of students served. For example, if students from families with higher incomes are more likely 
to apply to oversubscribed schools, the results of the impact study may not actually fully apply to the 
broader population of students who seek to enroll in pre-K and on average have lower levels of family 
income. Relatedly, the characteristics of the oversubscribed schools could differ substantially from 
the schools not represented in the lottery analysis, creating further challenges with generalizability 
across settings. 


All studies that use a randomized design—by leveraging naturally occurring lotteries or not—face 
these challenges. But, as evidenced by prior research, these issues may be exacerbated in lottery-based 
studies conducted in early childhood settings and must be handled carefully. 


FOCUS ON EARLY LEARNING: A UNIQUE OPPORTUNITY TO USE NATURALLY 
OCCURRING LOTTERIES TO STUDY THE EFFECTS OF A DISTRICT-WIDE 
ROLLOUT OF INSTRUCTIONAL ALIGNMENT ON STUDENT OUTCOMES 


As mentioned earlier, BPS uses a lottery approach to assign students to its public school pre-K 
program. This presents an opportunity to study the district-wide rollout of BPS’s Focus on Early 
Learning model to align instruction across pre-K and elementary school. Begun in a subset of schools 
in 2012, Focus on Early Learning was designed by BPS to adjust the structure of kindergarten, first 
grade, and second grade to closely mirror child-directed pre-K practices (such as small groups and 
play-based learning).'® The reform also added district-created components to the existing pre-K 
program, which centered around a combination of a language and literacy curriculum and a math 
curriculum created by experts in those learning domains.”° Pre-K themes and lesson topics are rein- 
troduced in higher grades, allowing content to be expanded in depth and complexity as children age. 
In this way, instructional alignment can allow for richer content and more cognitively demanding 
instruction in the higher grades. Box 1 provides a thematic example of how this works in the pre-K 
and kindergarten curricula. 


The BPS Department of Early Childhood created Focus on Early Learning because they recognized 
that their pre-K program—though found to have moderate to large favorable effects on student 
skills—was largely disconnected from children’s experiences in kindergarten through second grade. 
These grades did not effectively build upon the skills children were learning, potentially reducing the 
long-term impact of early childhood programming.”* Table 1 summarizes the key curricular compo- 
nents that make up Focus on Early Learning across both pre-K and kindergarten. 
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BOX 1 


EXAMPLE OF INSTRUCTIONAL ALIGNMENT IN 
PRE-K AND KINDERGARTEN 


e The Boston Public School district’s pre-K curriculum ends the year with a unit on things that grow. 
Vocabulary instruction on animal and plant names and related verbs is incorporated into the daily 
activities of the classroom. Sample vocabulary words that children learn include tadpole, cub, flock, 
hatch, and burrow. 


© Then, early in kindergarten, children participate in an animals and habitats unit, which introduces 
more complex vocabulary (for example, hibernate, life cycle, habitat, discover, and transform) and 
knowledge (for example, how animal habitats change across the season) that explicitly build on the 
introduction children learned in pre-K. 


TABLE 1 


OVERVIEW OF FOCUS ON EARLY LEARNING CURRICULAR COMPONENTS 
ACROSS PRE-K AND KINDERGARTEN 


CURRICULAR COMPONENT DESCRIPTION OF COMPONENT 

Introduction to centers A teacher identifies learning centers that are available to students and models 
some activities that children can engage in during center time. 

Centers Children use various media to communicate their ideas about a topic. Children 
are able to choose which center to join and are able to move freely between 
centers. 

Thinking and feedback When observing their work or the work of their classmates, children are 


prompted to look, notice, listen, wonder, suggest, and inspire. This helps 
develop independent thinking skills. 


Read aloud Students read a book from the curriculum four separate times with explicit 
goals and activities during each read. 


Literacy whole group Whole group time that is focused on phonics, phonological awareness, or 
other literacy topics (depending on grade). 


Literacy small group Small groups of children are led by the teacher to work on literacy-related 
material (e.g., reading, vocabulary, guided writing, word study). This work 
should overlap with work done on other components. 


Math whole group Whole group time that is focused on launching/introducing math concepts 
prior to the math workshop/small group. 


Math small group Children may do the same math activity that was introduced in the whole 
group or a different one. 


SOURCE: McCormick et al. (2020). 


NOTE: Curricular components are distinct activity types that occur regularly as part of the Focus on Early Learning curriculum. 
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The research team worked closely with the BPS district to develop an observational tool to examine 
implementation of Focus on Early Learning in pre-K and kindergarten classrooms. Results from this 
work, collected during the 2016-2017 and 2017-2018 school years prior to the pandemic, showed that 
implementation of the model was strong, with some variation across curricular components.?? The 
team also found evidence that Focus on Early Learning did support vertical alignment of children’s 
learning as they transitioned into kindergarten. Yet questions remain about whether the district-wide 
rollout of Focus on Early Learning has succeeded in improving students’ academic outcomes during 
elementary school. 


OVERVIEW OF THE STUDY SAMPLE 


Box 2 summarizes the data sources that the team used to achieve the aims explored in this brief. 

As noted, this study includes three cohorts of students who applied to a BPS pre-K program for the 
2012-2013 to 2014-2015 academic years. The team chose to include these cohorts of students in the 
study because they completed third grade by spring 2019, and thus have standardized test data avail- 
able for the outcome time points of interest. Although students might have participated in multiple 
lotteries because they applied to more than one oversubscribed school, the research group they were 
assigned to depended on the outcome of the first lottery in which they participated. As such, program 
group students are defined as those whose first pre-K lottery was for an aligned school (that is, a school 


BOX 2 
SUMMARY OF DATA SOURCES USED TO ACHIEVE RESEARCH AIMS 


e@ Application data were collected from BPS on all students who applied to pre-K for the 2012 to 
2013, 2013 to 2014, and 2014 to 2015 school years. The team used these data to identify lotteries 
for the pre-K program. 


@ Enrollment data were collected for the 2012 to 2013, 2013 to 2014, and 2014 to 2015 school years 
to identify where students enrolled in pre-K and kindergarten and how long they attended. 


e BPS data were collected on years that each school implemented Focus on Early Learning in both 
pre-K and kindergarten. The team used these data to select the lotteries for pre-K programs located 
in aligned schools. 


e Data were acquired from the State of Massachusetts on all students who applied to a BPS pre-K 
to further explore baseline equivalence. 


@ Publicly available data were collected on school-level demographic characteristics. 
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choosing to implement Focus on Early Learning in both pre-K and kindergarten) and who “won” a slot 
in that school. Students who “lost” their first pre-K lottery for an aligned school were assigned to the 
control group.?* This is similar to the approach used in prior work with the BPS pre-K lottery data.?* 


As described in Figure 1, of the full sample of pre-K lottery participants enrolled across three cohorts 
(N = 10,318), about 46 percent applied to an oversubscribed school and thus participated in a lottery. 
The remaining 54 percent of pre-K applicants did not participate in a lottery and received a spot in 

a school that was not oversubscribed. Among the students who participated in lotteries (N = 4,725), 
about 56 percent (N = 2,657) applied to an aligned school. That group—about a quarter of all pre-K 
applicants—is the student sample of interest in the current study. Within that group, about 30 per- 
cent of students won their first lottery and make up the program group. The remaining 70 percent of 
students lost their first lottery and form the control group. Students in the control group followed one 
of three paths: They enrolled in a different aligned school further down their choice list, enrolled in 
an unaligned school further down their choice list, or did not enroll in BPS pre-K at all. 


Table 2 summarizes information about BPS schools and how many implemented the Focus on Early 
Learning model in both pre-K and kindergarten. Of the 75 total elementary schools offering the BPS 
pre-K program during at least one year from 2012 to 2015, 48 (or 64 percent) implemented the Focus 
on Early Learning program in both pre-K and kindergarten in at least one of the study years. Of those 
schools, 46 conducted a lottery during at least one of the years studied. Thus, 96 percent of the total 
aligned schools are represented in the lottery impact sample for at least one year (the range across 
years is 83 percent to 98 percent). In addition, more than half of the total BPS schools offering pre-K 
are represented in the impact study for at least one cohort. The team then examined the extent to 
which lotteries were clustered within and across schools. Twenty schools (or 44 percent of all schools 
implementing Focus on Early Learning) were responsible for 75 percent of the lotteries. Although 
schools in the impact study have significant overlap with the broader sample of schools implementing 
the aligned model, there are a subset of school that have more representation in the study than oth- 
ers, a feature of the randomization process that will be further addressed in the final impact analysis. 


ISSUE 1: BASELINE EQUIVALENCE 


The first key issue that the research team investigated was whether the naturally occurring lotteries 
created program and control groups of students with similar characteristics (that is, that were equiv- 
alent) at the time of randomization. Researchers found that there was evidence of baseline equiva- 
lence along the student demographic characteristics available for the study sample. 


The research team arrived at these findings using two approaches, both of which are standard in edu- 
cation evaluation literature.?° The team looked at the following characteristics to assess whether they 
were equivalent for students in the program and control groups: a child’s age, race, home language, 
and country of origin; whether the child was female; whether they were eligible for a free or reduced- 
price lunch; and whether they were a dual language learner. This approach accounted for lottery 
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FIGURE 1 


BPS PRE-K APPLICANTS, LOTTERY PARTICIPANTS, 
AND PARTICIPANTS IN LOTTERIES FOR ALIGNED SCHOOLS 


N = 10,318 students apply to BPS pre-K for the 2012-2013 to 2014-2015 academic years 


N = 5,593 students prefer a pre-K that N = 4,725 students prefer an 
is NOT oversubscribed and do not oversubscribed pre-K program and 
participate in a lottery participate in a lottery 


N = 2,068 students prefer an N = 2,657 students prefer an 
oversubscribed pre-K program in an oversubscribed pre-K program in an 
unaligned school in their first lottery and aligned school in their first lottery and 
are excluded from current impact study make up the current impact sample 


N = 808 win their first lottery for an N = 1,849 lose their first lottery for an 
aligned school and make up the aligned school and make up the 
PROGRAM GROUP CONTROL GROUP 
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TABLE 2 
SUMMARY OF SCHOOL SAMPLE SIZES 


TOTAL NUMBER ALIGNED PERCENTAGE OF 

OF SCHOOLS ALIGNED SCHOOLS WITH ALIGNED SCHOOLS 

YEAR OFFERING PRE-K SCHOOLS LOTTERIES WITH LOTTERIES 
2012-2013 70 44 43 98 
2013-2014 71 47 40 85 
2014-2015 74 48 40 83 


Total 75 48 46 96 


NOTE: In this study, schools that implement Focus on Early Learning in both pre-K and kindergarten are 
described as aligned. 


blocks.?® Figure 2 presents results from this analysis. As illustrated, there were few differences in 
demographic characteristics between program and control group members except for the percentage 
of participants who were White. Eighteen percent of the program group was White compared with 
22 percent of the control group; this difference was statistically significant at the 5 percent level. There 
was a 3 percentage point difference between the number of program and control group students eli- 
gible for free and reduced-price lunches—a lower percentage of the program group was eligible—but 
this difference was not statistically significant. On September 1 of the pre-K year, the team examined 
the children’s ages and did not find any difference between the program (mean age = 4.52) and con- 
trol (mean age = 4.53) groups. 


As recommended by the U.S. Department of Education’s What Works Clearinghouse and examined 
in prior work with BPS lottery data, the research team then looked at whether there were systematic 
differences between the program and control groups when all student-level demographic characteris- 
tics were taken into account.?” The analysis, described in more technical detail in Box 3, determined 
that the school assignment process and resulting naturally occurring lotteries created equivalent 
groups prior to the start of pre-K. 


ISSUE 2: GENERALIZABILITY OF THE STUDENT SAMPLE 

The next key issue researchers wanted to understand was whether the students who participated in 
lotteries for aligned schools were representative of the broader population of students who applied to 
the BPS pre-K program. 

To fully investigate this issue, the team first compared the demographic characteristics of the full 


population of students who applied to BPS pre-K to the subsample of students who applied to schools 
that adopted the aligned approach. The next analysis compared both groups to the aligned schools 
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FIGURE 2 


DEMOGRAPHIC CHARACTERISTICS OF STUDENTS ASSIGNED TO 
THE PROGRAM AND CONTROL GROUPS 
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NOTES: Statistical significance levels are indicated as: ** = 1 percent; * = 5 percent. 
"FRPL-eligible" means that the student is eligible for free or reduced price lunches. 


BOX 3 
DETAILS ON ANALYSIS FOR EXAMINING BASELINE EQUIVALENCE 


To examine baseline equivalence in the impact study sample, the team used a regression analysis to 
examine the strength of the relationship between the indicator for whether a student was assigned to 
the program group and all of the observed demographic characteristics captured at enrollment and 
lottery blocks together. The F-test from the regression—which is a standard indicator of the strength of 
the relationship between the predictors and outcomes in a regression—was used to examine whether 
the characteristics on their own predicted whether students were assigned to the program or control 
group, when examined as a set. The F-statistic was not statistically significant (F (10,2298 = 1.31), 
p = .22), indicating that there were no systematic differences between the program and control groups. 
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lottery subsample included in the impact study—the students that competed in lotteries for aligned 
schools. As summarized in Figure 3, students who competed in lotteries for aligned schools were 
demographically similar to the group who applied to aligned schools, with some exceptions. For 
example, students who competed in lotteries for aligned schools were 7 percentage points less likely 
to be eligible for a free or reduced-price lunch, 4 percentage points less likely to be Hispanic, and 5 
percentage points less likely to be a dual language learner. The groups were also comparable with 
respect to child age, a characteristic not presented in Figure 3. Researchers conducted this compar- 
ison to mimic a generalizability analysis in a randomized controlled trial, where it is important to 
check that the students included in the impact analysis are representative of all students interested in 
enrolling in the program under study. 


FIGURE 3 


DEMOGRAPHIC CHARACTERISTICS OF ALL PRE-K APPLICANTS, 
APPLICANTS TO ALIGNED SCHOOLS, AND THE 
FULL LOTTERY SUBSAMPLE 
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NOTE: "FRPL-eligible" means that the student is eligible for free or reduced price lunches. 


The more policy-relevant question, however, is whether the students in the impact study were repre- 
sentative of the broader population of students who sought to enroll in any pre-K program and could 
potentially benefit from instructional alignment. There was also fairly good evidence of generalizabil- 
ity when making this comparison. Students participating in lotteries for aligned schools were 6 per- 
centage points less likely to be eligible for a free or reduced-price lunch and 3 percentage points less 
likely to speak Spanish at home, compared with the full population of students who applied to pre-K. 
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These were statistically significant differences. However, the students in the lotteries for aligned 
schools were representative of all pre-K applicants with respect to race, ethnicity, age, gender, and 
immigrant status. This comparison suggests that the ultimate results of the impact study can inform 
the BPS district more broadly about the impacts of instructional alignment on pre-K students. 


ISSUE 3: GENERALIZABILITY OF THE SCHOOL SAMPLE 


The final issue researchers investigated was whether the oversubscribed, aligned schools were repre- 
sentative of all schools that adopted Focus on Early Learning in pre-K and kindergarten and all BPS 
schools offering pre-K. 


The team found evidence that the schools included in the impact study were, on average, representa- 
tive of both of these broader populations of schools. Table 3 summarizes the characteristics of all BPS 
schools that offered pre-K. That group contained aligned schools that implemented Focus on Early 
Learning in both pre-K and kindergarten and schools that offered a business-as-usual curriculum. 
Schools that adopted alignment were generally representative of the broader group of schools offering 
pre-K. As shown in the first two columns of Table 3, aligned schools had slightly higher percentages 
of English language learners (defined by the state of Massachusetts as students whose first language 
is not English and who are unable to perform classroom work in English). Aligned schools also had 
slightly more Hispanic students, fewer White students, and 2 percent fewer students who scored at a 
proficient level on third grade state English/Language Arts (ELA) tests. Differences in teacher char- 
acteristics were minimal. 


Importantly, however, there were stark differences when directly comparing the characteristics of 
schools that chose to implement Focus on Early Learning across pre-K and kindergarten and those 
that did not. Aligned schools served higher proportions of non-White students and students who 
were English language learners or came from families with low incomes. There was a 6 percentage 
point difference in the proportion of students who scored at a proficient level on the state ELA test, 
with unaligned schools scoring higher. Differences in state math test proficiency rates were less 
glaring. Interview data with school district staff indicate that some schools—typically those serving 
students with lower standardized test scores—were strongly encouraged to implement the Focus 

on Early Learning model. This district-wide approach may have affected schools’ decisions to adopt 
Focus on Early Learning for pre-K and kindergarten.”* Future work will investigate how factors cor- 
related with whether schools adopted the aligned model—such as the percentage of English language 
learners within each school—affect the impacts of the district-wide rollout of alignment on student 
outcomes. Impact work will also investigate how the clustering of lotteries within a smaller number 
of schools—wherein 75 percent of lotteries took place within 44 percent of the schools implementing 
Focus on Early Learning—aftects the generalizability of study results. 
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TABLE 3 
CHARACTERISTICS OF BOSTON PUBLIC SCHOOLS THAT OFFER A PRE-K PROGRAM 


ALIGNED DIFFERENCE 


ALL SCHOOLS BETWEEN ALIGNED 
ALL ALL ALIGNED ~~ UNALIGNED REPRESENTED AND UNALIGNED 
CHARACTERISTICS SCHOOLS SCHOOLS SCHOOLS IN LOTTERIES SCHOOLS 
Student demographics (%) 
Students from families with low incomes? 73.0 75.0 69.4 73.8 5.6* 
Students from economically disadvantaged backgrounds? 50.5 52.1 47.6 54.0 4.6 
English language learner 34.1 37.9 27.0 38.4 10.9*** 
Non-English home language 42.5 46.3 35.6 46.9 10.8*** 
Students with disabilities 20.0 20.1 19.6 18.2 0.5 
Race 
Black 32.5 B13 34.2 30.5 -2.9 
Hispanic 42.9 45.4 38.3 46.6 7.1 ** 
White 15.1 13.4 18.4 13.3 -5.1* 
Asian 6.0 6.5 5.0 6.3 1.5 
Other race 3.6 3.4 4.1 3.3 -0.7 
Female 47.8 48.0 47.5 47.9 0.5 
Student academic performance on 3rd grade standardized tests (%) 
Scored at a proficient level on ELA exam 36.6 34.5 40.3 33.1 -5.9* 
Scored at a proficient level on math exam 48.0 47.3 49.1 47.6 -1.8 


(continued) 
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TABLE 3 (continued) 


ALIGNED DIFFERENCE 

ALL SCHOOLS BETWEEN ALIGNED 

ALL ALL ALIGNED UNALIGNED REPRESENTED AND UNALIGNED 

CHARACTERISTICS SCHOOLS SCHOOLS SCHOOLS IN LOTTERIES SCHOOLS 
Teacher and school characteristics 

Licensed teachers (%) 96.7 97.1 96.0 97.1 11 

Number of students per teacher 13.59 13.51 13.79 13.76 -0.3 

Exemplary teachers (%) 17.3 16.8 18.5 16.1 -1.7 

Proficient teachers? (%) 77.0 77.8 75.4 77.6 2.3 

Teacher retention® (%) 75.6 76.2 74.3 77.9 1.9 

Stability’ (%) 87.6 87.6 87.7 87.9 -0.1 

Average number of students per class 17.99 18.02 18.02 18.32 0.0 


NOTES: *An indication that the student is either eligible for free or reduced-price lunch or that the student receives Transitional Aid to Needy Families benefits. Data were 
available in years 2012-2013 and 2013-2014. 

>Calculated based on a student's participation in one or more of the following state-administered programs: the Supplemental Nutrition Assistance Program; the Transitional 
Assistance for Families with Dependent Children; the Department of Children and Families’ foster care program; and MassHealth (Medicaid). Data were available for schools 
in 2014-2015. 

°A rating of exemplary is defined as a performance that is of such a high level that it could serve as a model. 

4A proficient performance rating is defined as being fully satisfactory. 

“This represents the percentage of teachers retained throughout the school year. 

The stability rate measures how many students remain in in a district or school throughout the school year. 
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FURTHER ISSUES TO CONSIDER 


Although these findings are encouraging and suggest that the impact study has met key criteria 
needed to conduct rigorous, policy-relevant research, there are further methodological issues that the 
team will address as part of the ultimate impact study. The first is to clearly define the intervention 
that the team is testing. As noted above, schools in this study were not randomly assigned to imple- 
ment the Focus on Early Learning model. Rather, they chose to adopt the model on their own and the 
research team found that there were school-level characteristics associated with implementation of 
the aligned approach. Accordingly, it is critical to remember that this study is only able to estimate 
the impacts of a district-wide rollout of an instructional alignment approach across grades, including 
the characteristics associated with schools’ decision to adopt Focus on Early Learning. This study can- 
not estimate the unique causal impact of instructional alignment across pre-K and kindergarten over 
and above the factors associated with whether schools choose to adopt the model. Further investiga- 
tion of these factors will be a critical part of future work. 


Second, the team will investigate the differences in the instructional experiences of students in 

the program and control groups across both pre-K and kindergarten. This study seeks to compare 
schools that adopted the Focus on Early Learning approach in pre-K and kindergarten with schools 
that did not implement this approach. In other words, students in the control group typically still 
attended pre-K and kindergarten, but most did not attend a public school or other pre-K setting that 
chose to adopt the Focus on Early Learning curricula and professional development model across 
both grades. The method of quantifying the difference in program and control group students’ expe- 
riences in pre-K and kindergarten will echo how prior research tested the impacts of a new curricu- 
lum or instructional enhancement.?® Due to the retrospective nature of the study, however, the team 
lacks fine-grained observational data to describe the extent to which instruction differed for students 
in the program group compared with those in the control group. To address this challenge, future 
work will describe the full range of curricula that students experienced in both pre-K and kindergar- 
ten in order to quantify in more detail the differences in instruction across groups. 


The team will also fully explore compliance, or the extent to which students assigned to the program 
group enrolled in an aligned school and students assigned to the control group did not. In other 
words, compliance captures the proportion of students who did indeed comply with the instruc- 
tional experience to which they were randomly assigned. Compliance will be critical for estimating 
the impact of enrollment in an aligned school on the student outcomes of interest, such as academic 
skills in third grade. 


Relatedly, this study is constrained by the outcome measures, which are only available via adminis- 
trative records in third grade. Complementary research on pre-K fadeout and convergence suggests 
the need for outcome data collected in kindergarten, first, and second grade that capture the full 
range of key skills children develop in pre-K and throughout elementary school.*° To address this 
limitation, researchers will need to explore different approaches for measuring a broader range of 
children’s skills over time with the available outcome data. The team may also encounter issues with 
sample attrition if students enrolled in the study leave the state of Massachusetts and do not have any 
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available outcome data. This would be a particularly challenging situation if attrition is more likely 
for the control group than the program group or vice versa. Attrition posed minimal problems in a 
previous lottery-based study of Boston pre-K but this issue will be carefully examined in the impact 
study. 


IMPLICATIONS OF THE FINDINGS 


The implementation of Focus on Early Learning—coupled with the existence of naturally occurring 
lotteries for oversubscribed schools—presents an unprecedented opportunity to estimate the causal 
impacts of a district-wide rollout of instructional alignment across pre-K and kindergarten on stu- 
dents’ learning and development over time. This research brief provides evidence that there is a good 
foundation upon which to conduct this study. 


e First, there is evidence that random assignment worked for the three cohorts of students who 
applied for slots in aligned schools: Prior to the start of pre-K, students assigned to the program 
group were demographically similar to students assigned to the control group. 


e Second, the group of students competing in lotteries for aligned schools were representative of 
the broader group of students who applied to pre-K. This is important because it means that the 
results from the impact study can help the district understand the potential impact of the align- 
ment rollout for the full group of students seeking entrance to the public pre-K program. 


e Finally, there is promising evidence that the schools in the impact study are representative of the 
broader population of schools offering pre-K. Even so, given larger differences between aligned 
and unaligned schools, there is a clear need to further examine the schools’ characteristics—such 
as the percentage of English language learners enrolled—that are potentially correlated with adop- 
tion of Focus on Early Learning. 


Having found encouraging evidence that these data can indeed be used to build causal and general- 
izable evidence for BPS on the rollout of its vertical instructional alignment approach across pre-K 
and kindergarten, the research team will next link these data to information on student outcomes: 
children’s ELA and math standardized test scores through third grade, receipt of special education 
services, attendance, persistence in BPS public schools, grade retention, and literacy skills in ele- 
mentary school. The team will build off the best evidence in the field to estimate impacts on student 
outcomes through third grade.*" Results will directly inform policymakers and practitioners in 
Boston and beyond about the value of rolling out a district-wide curriculum like Focus on Learning 
to sustain the pre-K boost. 
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